Skip to content

Fix the deletion of S3 buckets - #14678

Open
bernardodemarco wants to merge 1 commit into
apache:4.22from
scclouds:fix-object-store-allocated-size
Open

Fix the deletion of S3 buckets#14678
bernardodemarco wants to merge 1 commit into
apache:4.22from
scclouds:fix-object-store-allocated-size

Conversation

@bernardodemarco

@bernardodemarco bernardodemarco commented Nov 11, 2026

Copy link
Copy Markdown
Member

Description

PR #10140 introduced the allocated_size column to the cloud.object_store table, which is expected to hold the sum of all bucket quota, in bytes. To initialize the new column, the following query was introduced:

cloudstack/engine/schema/src/main/resources/META-INF/db/schema-42040to42100.sql

Lines 742 to 749 in f8f3848

UPDATE `cloud`.`object_store`
JOIN (
SELECT object_store_id, SUM(quota) AS total_quota
FROM `cloud`.`bucket`
WHERE removed IS NULL
GROUP BY object_store_id
) buckets_quota_sum_view ON `object_store`.id = buckets_quota_sum_view.object_store_id
SET `object_store`.allocated_size = buckets_quota_sum_view.total_quota;

However, bucket quotas are stored in GiB, not bytes, and the query does not convert them. As a result, Object Storage providers in environments upgraded from versions prior to 4.21 have an allocated_size that is smaller than expected. When a bucket is deleted, its quota (converted to bytes) is subtracted from this value, which becomes negative. Since the column is unsigned, the update fails and the bucket cannot be deleted:

Impossibility of deleting buckets 
2026-11-10 16:10:07,939 ERROR [o.a.c.s.d.d.ObjectStoreDaoImpl] (qtp1404565079-3736:[ctx-3a285296, ctx-ce4f3b14]) (logid:78d41388) DB Exception on: HikariProxyPreparedStatement@1008598898 wrapping com.mysql.cj.jdbc.ServerPreparedStatement[21]: UPDATE object_store SET object_store.allocated_size=-536870912000 WHERE object_store.id = 2 com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Out of range value for column 'allocated_size' at row 1
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:104)
	at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
	at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)

Therefore, this PR recalculates the allocated size in bytes for each Object Storage provider. It targets the 4.22.2.0 version. A follow-up PR targeting main will apply the same fix to environments already running 4.23.0.0.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

  • Executed the normalization query locally and verified it recalculates correctly the allocated size for each Object Storage provider.

@bernardodemarco bernardodemarco added this to the 4.22.2 milestone Nov 11, 2026
@bernardodemarco bernardodemarco mentioned this pull request Nov 11, 2026
Open
2 of 12 tasks
@codecov

codecov Bot commented Nov 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 18.32%. Comparing base (b56bbc2) to head (74017f5).

Additional details and impacted files 
@@            Coverage Diff            @@
##               4.22   #14678   +/-   ##
=========================================
  Coverage     18.32%   18.32%           
- Complexity    16644    16647    +3     
=========================================
  Files          5939     5939           
  Lines        537787   537787           
  Branches      65932    65932           
=========================================
+ Hits          98544    98553    +9     
+ Misses       428033   428023   -10     
- Partials      11210    11211    +1
Flag Coverage Δ
uitests 4.08% <ø> (ø)
unittests 19.41% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@winterhazel winterhazel Awaiting requested review from winterhazel
@hsato03 hsato03 Awaiting requested review from hsato03

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

4.22.2

Development

Successfully merging this pull request may close these issues.

1 participant

@bernardodemarco